\section{Problem Space}

\subsection{Trace Files}



\begin{table*}[htdp]
\caption{...}
\begin{center}
\begin{tabular}{llrrrrr}
\toprule
{\bf Source} & {\bf Time} & {\bf Procs} & {\bf Spawns} & {\bf Dies} & {\bf Sends} & {\bf Receives} \\
\midrule

Ericsson & 14s  & 1810 & 1723 & 1570 & 18708 & 29034 \\
CouchDB  & 46s  & 2211 & 2020 & 2179 & 39611 & 98256 \\
Tsung    & 102s & 724  & 0    & 212  & 12654 & 16263 \\
Yaws     & 64s  & 190  & 38   & 162  & 2346  & 4548 \\
Router   & 91s  & 984  & 14   & 951  & 2985  & 5728 \\

\bottomrule
\end{tabular}
\end{center}
\label{default}
\end{table*}



The behaviour of Erlang such as function calls, process states and process communication can be recorded during runtime using parts of the debug module dbg. This module offers a variety of tracing functions such as dbg:trace and dbg:p as well as others for different purposes.
We collected trace files of six large-scale Erlang systems.  
\begin{enumerate}
\item{\emph{CouchDB}. Apache CouchDB is a distributed, fault-tolerant, nonSQL database server written in Erlang. The trace file was obtained by tracing a debugging shell with an Apache CouchDB database set up on localhost and running all of the provided test cases.}
\item{\emph{Yaws}. Yaws is a HTTP high performance webserver that is appripriate for hosting dynamic-content webapplications written in Erlang for lightweight concurrency. The trace was gathered by running the server on localhost and browsing through the pages.}
\item{\emph{Tsung}. Tsung is a distributed multi-protocol load testing tool for stress testing webservers supporting databases or otherwise. The trace file was generated by creating one hundred virtual clients trying to connect and chat on an Ejabberd server (see below).}
\item{\emph{Ejabberd}. Ejabberd is a cross-patform Jabber instant messaging server written in Erlang for multithreading ability. The trace file was generated by simulating a instant messaging conversation using Tsung.}
\item{\emph{Telephony Exchange Server}. Ericsson provided a runtime trace of a telephony exchange server that acts as a translator between the Ericsson proprietary protocol and H.248 protocol used by common media gateways.}
\item{\emph{Router}. Francesco Cesarini, a highly involved Erlang developer provided a runtime trace of an automated test case running on a router node. The router is part of the launching of a new wireless Instant Messaging gateway.}
\end{enumerate}

The tracing was set up to look at all the existing processes during runtime, with the exception of the Ericsson and Router traces, which were obtained from outside sources [<ref>]. The tracer was set up on a linked Erlang shell during the runtime of our test software, and the results were written to file: 
\begin{mylisting}
\begin{verbatim}
dbg:tracer(port, dbg:trace_port(file, "File.trace")).
dbg:p(all, [timestamp, procs, m]).
\end{verbatim}
\end{mylisting} 
Initially all of our traces used the flags \texttt{new} and \texttt{self()} were used in place of \texttt{all}. This was set up so that we capture only software related processes, while abstracting background system interaction. However, this became a problem when tracing software during runtime without restarting the whole system. 

\begin{comment}
The specific behavior of an Erlang program may be recorded during runtime using the erlang:dbg module. This module offers variety of functions for tracing such as dbg:trace and dbg:p. By closely analyzing software written in Erlang, it can enhance the general understanding of behavior pattern found in different large-scale distributed system. However, the information gathered by these functions is not readable for an inexperienced user, especially when investigating large-scale distributed software.

We propose to design a visual tool for Erlang trace analysis. However, it is important to analyze the traces before actual design of the tool. Prior analysis of traces will be beneficial for understanding of process role and process behavior. Any messaging or spawning pattern between the processes will be helpful in design of the tool. We have some observations based on the initial analysis of traces.
\end{comment}


\subsection {What is the Distribution of Spawning Activity?}

The spawning pattern in analyzed Erlang traces is inconsistent. However, a common observation is that most spawning is usually done by a very small number of processes as compared to total number of processes.  From Table  \ref{spawn_table} we can observe that all analyzed Erlang traces have a very large number of processes out of which a very few processes spawn. 

Figure \ref{fig:spawn1} depicts spawning patter of top 20 processes in CouchDB, Tsung and Yaws. Processes are sorted in ascending order of their total number of spawns. We observe that maximum spawning is done by top most processes. For example, in CouchDB, 97.49\% of total spawning is done by top 10 processes whereas, 66 processes out of 76 spawns only once. Similarly, in Yaws, 68.40\% spawning is by top 10 processes and 82 out of 89 processes spawns only once. We also observe some exception in this pattern, in Tsung, 99.52\% of total spawning is obtained by a single processes and ej\_ts and ej\_2usr have only 1 and 5 spawns respectively. 

\begin{figure}[h!]
\centering
\psfig{file=spawn.png, scale=0.35}
\caption{Spawning pattern of top 20 processes in CouchDB}
\label{fig:spawn}
\end{figure}


\begin{table}[htdp]
\caption{Spawning}
\begin{center}
\begin{tabular}{|c|c|p{2 cm}|p{1.5 cm}|}
\hline
Trace file &\# of processes & \# of spawning processes & \# of spawns\\
\hline
couchdb & 235123 & 76 & 2634\\ \hline
yaws & 20769 & 89 & 250\\ \hline
tsung & 68797  & 6 & 1255\\ \hline
ej\_2usr & 1237 & 3 & 5 \\ \hline
ej\_ts & 418483 & 1 & 1 \\ \hline

\end{tabular}
\end{center}
\label{spawn_table}
\end{table}%

\subsection {What is the Distribution of Functions Being Spawned? Do Processes Tend to Spawn A Small Number of Functions?}

\subsection {What is the Distribution of Message Sending Activity?}

\begin{figure}[h!]
\centering
\psfig{file=sends.png, scale=0.39}
\caption{Message sending distribution of the first 20 actors in each trace}
\label{fig:send1}
\end{figure}


In general, message sending patterns are unique for each piece of software. However, we observed recurring baseline behaviors in message sending in our Erlang traces as depicted in Figure \ref{fig:send1}. By sorting the processes depending on the percentage of total sends we found that there is hierarchal distribution. In other words, there are several groups of processes in which there are similar numbers of sends.

We observed that within the first few groups of actors, most of the message sending is initiated by small number of processes. For example, in Tsung, we observe as little as 10 of the 1278 actors engage in more than 70\% of total message sending and similarly, in CouchDB, 20 of the 2667 actors engage in over 60\%.



\subsection {Are Messages Consumed in the Order they are Sent? What is the Delay?}

\begin{table}[htdp]
\caption{Send-Receive Delay}
\begin{center}
\begin{tabular}{|c|p{1 cm}|c|p{1 cm}|p{1.2 cm}|}
\hline
Trace file &  Max Delay & Total Delay & Delay & Events / sec\\
\hline
tsung  & 943 & 391001 & 1.96 \% &  3444 \\ \hline
couchdb  & 2056 & 542570 & 0.67 \% & 2912 \\ \hline
yaws & 981 & 48970 & 0.22 \% & 917\\ \hline
ej\_2usr & 111 & 6992 & 0.02 \% & 32 \\ \hline
ej\_ts  & 225 & 2648 & 0.01 \% & 18 \\ \hline
\end{tabular}
\end{center}
\label{spawn_table}
\end{table}%

The Erlang concurrency model supports asynchronous message sending. Therefore, once a message is sent, it is independent of any other actions the sender or receiver participate in. Because of this, message receiving patterns can prove to be quite unpredictable. Upon close inspection of the Ejabberd trace, we see that the maximum delay occurs when sending messages to the system process 0.4.0 which stands for "error\_logger". On the other hand, for the Tsung trace, the longest delay occurs when the most important process 72 sends a message to the second most highly ranked process, 71. The pattern that is evident is that the more involved an actor, the higher the delay for receiving messages. Furthermore, the total delay is directly dependent on the activity (events per second) of a specific trace.

\subsection {How Coupled Are Processes to their Parent/Siblings?}

\subsection {How Long Lived Are Processes?}
